{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon FSx for Lustre is a high-performance file system optimized for deep learning workloads.  \n",
    "\n",
    "FSx provides POSIX-compliant file system access to S3 for multiple readers and writers simultaneously."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Install the FSx CSI Driver for Kubernetes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "source ~/.bash_profile\n",
    "\n",
    "kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/release-0.2.0/deploy/kubernetes/manifest.yaml \n",
    "\n",
    "#### Get VPC ID\n",
    "export VPC_ID=$(aws ec2 describe-vpcs --filters \"Name=tag:Name,Values=eksctl-${AWS_CLUSTER_NAME}-cluster/VPC\" --query \"Vpcs[0].VpcId\" --output text)\n",
    "echo \"export VPC_ID=${VPC_ID}\" | tee -a ~/.bash_profile\n",
    "\n",
    "#### Get Subnet ID\n",
    "export SUBNET_ID=$(aws ec2 describe-subnets --filters \"Name=vpc-id,Values=${VPC_ID}\" --query \"Subnets[0].SubnetId\" --output text)\n",
    "echo \"export SUBNET_ID=${SUBNET_ID}\" | tee -a ~/.bash_profile\n",
    "\n",
    "\n",
    "# THIS MUST HAPPEN HERE BECAUSE VPC_ID IS DEFINED IN THIS CELL!!\n",
    "aws ec2 create-security-group --group-name eks-fsx-security-group4 --vpc-id ${VPC_ID} --description \"FSx for Lustre Security Group\" --query \"GroupId\" --output text\n",
    "\n",
    "#aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=${VPC_ID}\" --query \"Subnets[0].SubnetId\" --output text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "source ~/.bash_profile\n",
    "\n",
    "!aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=${VPC_ID}\" --query \"Subnets[0].SubnetId\" --output text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws ec2 create-security-group --group-name eks-fsx-security-group --vpc-id ${VPC_ID} --description \"FSx for Lustre Security Group\" --query \"GroupId\" --output text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Create your security group for the FSx file system\n",
    "export SECURITY_GROUP_ID=$()\n",
    "echo \"export SECURITY_GROUP_ID=${SECURITY_GROUP_ID}\" | tee -a ~/.bash_profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Add an ingress rule that opens up port 988 from the 192.168.0.0/16 CIDR range\n",
    "aws ec2 authorize-security-group-ingress --group-id ${SECURITY_GROUP_ID} --protocol tcp --port 988 --cidr 192.168.0.0/16\n",
    "\n",
    "#### Update the environment variables in the storage class spec file\n",
    "# Populate SUBNET_ID, SECURITY_GROUP_ID, S3_BUCKET\n",
    "cd ~/SageMaker/kubeflow/notebooks/part-3-kubernetes/\n",
    "\n",
    "sed \"s@SUBNET_ID@$SUBNET_ID@\" fsx/fsx-s3-sc.yaml.template > fsx/fsx-s3-sc.yaml\n",
    "\n",
    "sed -i.bak -e \"s@SECURITY_GROUP_ID@$SECURITY_GROUP_ID@\" fsx/fsx-s3-sc.yaml \n",
    "\n",
    "sed -i.bak -e \"s@S3_BUCKET@$S3_BUCKET@\" fsx/fsx-s3-sc.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Setup IAM Policies to allow Worker Nodes to Access FSx and S3\n",
    "export POLICY_ARN=$(aws iam create-policy --policy-name fsx-csi --policy-document file://./fsx/fsx_lustre_policy.json --query \"Policy.Arn\" --output text)\n",
    "echo \"export POLICY_ARN=${POLICY_ARN}\" | tee -a ~/.bash_profile\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aws iam attach-role-policy --policy-arn ${POLICY_ARN} --role-name ${INSTANCE_ROLE_NAME}\n",
    "\n",
    "#### Deploy the StorageClass and PersistentVolumeClaim\n",
    "kubectl create -f fsx/fsx-s3-sc.yaml\n",
    "kubectl create -f fsx/fsx-s3-pvc.yaml\n",
    "\n",
    "# You can check the status by running the following command (as many times as you like). \n",
    "\n",
    "# *Note:  This will take 5-10 minutes, so please be patient!*\n",
    "\n",
    "kubectl get pvc fsx-claim \n",
    "\n",
    "# Please wait for this to complete before continuing.\n",
    "\n",
    "# When the claim is bound, you will see the following:\n",
    "\n",
    "kubectl get pvc fsx-claim\n",
    "\n",
    "### EXPECTED OUTPUT ###\n",
    "# NAME        STATUS   VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE\n",
    "# fsx-claim   Bound    pvc-xxx                                    1200Gi     RWX           fsx-sc         1m\n",
    "\n",
    "echo \"Completed\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Monitor and Verify FSx File System Creation\n",
    "\n",
    "In the AWS Console, navigate to `FSx` > `File systems`\n",
    "\n",
    "![FSx File System](img/fsx-lustre.png)\n",
    "\n",
    "_Note: The minimum size of an FSx Lustre file system is 1.2 Terabytes._\n",
    "\n",
    "Please continue to the next section while the FSX Lustre file system is spinning up.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
